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ABSTRACT: This study was undertaken to investigate the progress in operational 
scientific literacy skills through demonstrating cognition associated with 
undertaking scientific processes. Scientific literacy is taken here to mean utilising 
science knowledge and skills, particularly with relevance to creative problem 
solving and making reasoned decisions in real life situations. The instrument was 
compiled based on a review of relevant international literature, plus competences 
emphasised in the Estonian science curriculum. Items in the instrument followed 
the levels of the Structure of Observed Learning Outcomes (SOLO taxonomy) and 
all tasks in the instrument were contextualized to relate to real life. A stratified 
sample of students from 44 schools gave a data set with returns from 1128 
beginning of 10th grade students and 946 end of 11th grade students. Lindings 
suggested that there was no sufficient shift from one grade to another in operational 
skills such as giving scientific explanations, problem solving and justified decision 
making. Although students did gain more knowledge and the expectation was that 
students would enhance their operational skills for scientific literacy, this was not 
reflected in their competence to solve problems. This suggested there was a need 
to re-think science teaching and learning approaches as well as the manner in which 
science was introduced to students. 

KEY WORDS: Scientific literacy, context-based tasks, curriculum-related 
progress, interdisciplinarity, SOLO taxonomy 


Introduction 

It is generally suggested that the purpose of science education is to promote 
scientific literacy (e.g. Murcia, 2009; EURYDICE, 2011; Soobard & 
Rannikmae, 2011; Aikenhead et al., 2011). While noting that a variety of 
definitions of Scientific and Technological Literacy (STL) exist, (e.g. 
OECD, 2007; Roberts, 2007), in the current study, scientific literacy is 
taken to mean utilising science knowledge and skills, particularly with 
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relevance to creative problem solving and making reasoned decisions in 
real life situations (Holbrook and Rannikmae, 2009). In this study such a 
definition is operationalised through three scientific literacy skills - using 
scientific evidence for problem solving, giving scientific explanations, and 
making reasoned socio-scientific decisions. These components are in line 
with multiple scientifically literate definitions focusing on cognitive 
component in scientific literacy (e.g. OECD, 2007; Sadler & Zeidler, 2005; 
Soobard & Rannikmae, 2011; Soobard & Rannikmae, 2014) and also 
included in the Estonian national science curriculum for upper secondary 
school students, as part of expected learning outcomes (Estonian 
Government, 2011). 

Unfortunately, there is no clear approach to the measurement of these 
operational skills as a component of scientific literacy, because what is and 
how it is assessed, is determined by what is valued by science education 
policy, the curriculum and the role of science teachers in delivering the 
content of the curriculum to students (Corrigan et al., 2013). In Estonia, 
assessment of science learning is mainly undertaken by means of testing 
and examination (Holbrook, 2008) with the assessment against norms put 
forward by education policy and curriculum developers, and therefore tends 
to be content focused. 

Biggs (1996) proposed using the Structure of Observed Learning 
Outcomes (SOLO) taxonomy for monitoring students’ curriculum related 
progress, as the SOLO taxonomy can be used to describe how a learner’s 
performance has grown in complexity and level of abstraction in moving 
from a uni-structural response to evidentially putting forward responses at 
an extended abstract level. SOLO taxonomy can be used for developing 
items requiring growth of knowledge as well as items requiring a 
demonstration of operational skills, e.g. students’ active participation 
during assessment, not simply remembering and re-producing learned 
science content. In other words, this approach goes beyond assessment for 
tests and examinations that focus on school science content. 

Another important assessment component is the assessment context. It 
seems desirable that instruments for measuring progress in scientific 
literacy be connected with real life issues (Teppo & Rannikmae, 2008; 
Holbrook & Rannikmae, 2009) e.g. context-based (Bennet et al., 2007) and 
not focusing simply on science content (OECD, 2007; OECD, 2014). 

The Purpose and Research Questions 

The puipose of this study is to assess students’ curriculum related progress 
at the end of grade 11 compared to measures at the beginning of grade 10 
in operational scientific literacy skills, approached via a real life societal 
context. The instrument, based on the SOLO taxonomy, seeks to identify 
students’ curriculum related progress at the gymnasium level (grades 10 - 
11), through their capability to transfer both knowledge and skills to new 


264 



situations, seen as important for enhancing scientific literacy. The 
expectation is that students at higher grade levels are better able to answer 
the questions posed, based on their increased operational skills in problem 
solving, explaining phenomena scientifically and making decisions. All 
these are contributing to raising the level of a person’s scientific literacy. 
The following research questions are posed: 

1. How do grade 10 and 11 students demonstrate their operational 
scientific literacy skills when responding to a context-based situation? 

2. What progress is associated with science learning in identified 
components of students’ scientific literacy after two years of study at 
the gymnasium level? 

Review of Relevant Literature 


Scientific literacy 

Scientific literacy has multiple definitions as illustrated in Table 1. The 
assessment of operational skills as part of scientific literacy has been 
moving towards testing using real life contexts, e.g. contexts related to real 
life, allowing students to apply the actual skills needed in the situation 
(OECD, 2007; OECD, 2014; Soobard & Rannikmae, 2011; 2014; Fensham 
& Rennie, 2013). According to Roberts (2007; 2011), these ideas are more 
related to the Vision 11 of scientific literacy, where the starting point for 
science learning is a familiar situation for students and testing of the 
relevant science knowledge and skills follows. In the case of school science, 
the emphasis is on promoting science knowledge that helps students to link 
school science and everyday life situations and help them to become 
informed users of the knowledge in their everyday lives (Sarkar & Corrigan, 
2014). 

Comparisons between components associated with scientific literacy 
shows that (see Appendix) scientific literacy has been broadly associated 
with knowledge (interdisciplinary), operational skills, nature of science and 
perception components. Focusing on the cognitive components means that 
a scientifically literate person needs an interdisciplinary understanding and 
concepts relevant to real-life situations, as well as operational skills 
important for solving problems and making decisions (Miller, 1998; Norris 
& Phillips, 2003; Millar, 2006; OECD, 2007; Holbrook & Rannikmae, 
2007; Choi et al., 2011; Soobard & Rannikmae, 2011; 2014). Also, 
although scientific knowledge is needed for being scientifically literate, it 
needs to be functional knowledge, ready to apply in unfamiliar situation 
(Rennie, 2011). This is also on the line with the PISA 2015 scientific 
literacy definition (OECD, 2013), which states that a scientifically literate 
person should demonstrate competences to explain phenomena 
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scientifically, evaluate and design scientific inquiry and interpret data and 
evidence scientifically. An additional interpretation can be that is there is a 
need to avoid simply learning isolated facts, laws and concepts, and pay 
attention to interrelating them in a meaningful context, for example, in 
dealing with socio-scientific issues (Holbrook & Rannikmae, 2007). 

A further aspect considered integral to scientific literacy, indicated in 
table 1, is problem solving. Problem solving is a purposeful activity 
involving deliberate or controlled actions, not totally reliant on automatic 
processes. A problem only exists when the person trying to solve it lacks 
relevant knowledge to produce an immediate solution (Eysenck, 2012). 
However, there is no one single type of problem solving technique and 
different types of problems require different cognitive skills, e.g. from 
story-telling problems to design problems (Jonassen, 2011). In the current 
study, the focus is on science-related problems and solving such problems 
means finding a scientific solution (Holbrook & Rannikmae, 1997; 
Rannikmae, 2008; Soobard & Rannikmae, 2011). 

Decision-making, based on socio-scientific reasoning, is a further 
component associated with scientific literacy. According to Eysenck 
(2012), decision-making involves selecting from among several 
possibilities, e.g. it involves making a selection from various possible 
options. Laius (2011) suggests that the decision can also depend on the 
importance of various non-scientific factors, such as cost (economics), 
ethical and moral aspects, values, environmental concerns, etc. In the 
current study, decision-making with socio-scientific reasoning involves 
making a choice between several options and then giving reasons for the 
choice selected. It is seen as important to include this type of task in the 
assessment instrument, because, according to Sadler (2004) and Zeidler et 
al. (2005), students possess poor links between the science taught and its 
impact on society. They suggest that teaching/learning materials need to 
encompass a socio-scientific, issue-based approach. According to these 
authors, socio-scientific issues are difficult for students, because they are 
open-ended, ill-structured and include multiple decisions. At the same time, 
these issues can be promoted within a philosophy of “education through 
science” (Holbrook & Rannikmae, 2007), because it may include, 
according to Lee and Grace (2012), thinking in multiple directions (not 
focusing only on science) and encompassing moral judgments. Reasoning 
within socio-scientific issues has been associated with the process of 
informal, rather than formal, reasoning (Sadler, 2004), where informal 
reasoning involves a person’s arguments based on one's relevant 
knowledge and experiences (Eysenck, 2012). In other words, it is possible 
to make socio-scientific decisions based on personal experiences and not 
necessarily including reference to the appropriate formal scientific 
knowledge. 
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The science curriculum and science teachers in Estonia 


The Estonian science curriculum at the gymnasium (upper secondary) level 
was investigated to ensure that the development of the test instrument is in 
line with expected learning outcomes in gymnasium level. The National 
Curriculum of Estonia for upper secondary school indicates competences 
through the following learning outcomes, which should be promoted in 
each science subject (Estonian Government, 2011): 

1. Solving problems using scientific methods (e.g. posing scientific 
questions, planning scientific investigations, controlling variables, 
analysing and interpreting results, drawing conclusions). 

2. Making reasoned decisions taking into account scientific, 
environmental, social, economic, political and ethical-moral 
considerations. 

3. Using and relating (referring to interdisciplinary) knowledge from 
Biology, Geography, Chemistry and Physics subjects for problem 
solving and decision-making. 

The Estonian science curriculum includes four compulsory science 
subjects (chemistry, biology, geography and physics) at the upper 
secondary school level and all, accordingly to the puiposes of these science 
curricula, are expected to contribute to enhancing scientific literacy. Until 
2012, it was compulsory for grade 12 students to take national examinations 
and it was common that students sat at least some examinations in science 
subjects. An analysis of the latest state examination items showed that 
geography items followed Bloom's taxonomy (National examination 
commission (Geography), 2013), while in biology, the focus was on the 
content knowledge and its applications in real life (e.g. interpreting tables, 
schemes, graphs, relations applications in everyday life, drawing 
conclusions, transferring knowledge into a new context) (National 
examination commission (Biology), 2013). In chemistry and physics, the 
main focus was on science content (Klooster, 2013; National examination 
commission (Chemistry), 2013). 

In Estonia, a Teacher Needs Questionnaire (TNQ) was administered 
among 30 experienced gymnasium science teachers. Findings from this 
study showed that science teacher’s self-confidence was high, for example, 
in interdisciplinary teaching, but lower in promoting inquiry-based 
learning, putting forward the goals of education and in assessing students. 
However, their self-perceived training need was high in all subscales (e.g. 
assessment, classroom learning environment, motivation, inquiry based 
learning) (Holbrook, Rannikmae & Valdmann, 2014). 

Even more, when gymnasium students in Estonia were asked to 
indicate their perceptions against received science teaching, they did not 
perceive that science subjects (especially chemistry and physics) were 
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expected to support either the development of problem solving and decision 
making skills, or other components of scientific literacy (Soobard & 
Rannikmae, 2014). Only the development of interdisciplinary knowledge 
was highlighted and this was also the field where science teachers seemed 
to have higher perceived self-confidence (Holbrook, Rannikmae & 
Valdmann, 2014). Based on an analysis of student perceptions, operational 
skills do not appear to be promoted in a sufficient manner (Soobard & 
Rannikmae, 2014). 

Context-based learning 

Cognitive processes in schools focus more on simple algorithms than their 
use in multiple everyday contexts (for example, one learns to use the 
algorithm with speed, but not its applications in varying situations) (Biggs, 
1996). At the same time, everyday situations are concerned with personally 
valued content and experiences in context. Therefore, it is suggested to use 
context-based items to assess students’ learning (Bennet et al., 2007; 
Fensham & Rennie, 2013). According to research (Bennet et al., 2007; 
Rannikmae, 2008; Feinstein, 2010; Fensham & Rennie, 2013; Sormunen, 
Keinonen & Holbrook, 2014), using context-based items to assess students 
learning: 

• influences students’ interest and motivation to answer;. 

• provides a deeper sense of students’ conceptual understanding; 

• promotes transfer of science learning to real life situations; 

• allows students to see the usefulness of their own knowledge in real 
life situations; 

• allows students to apply competences in a context derived from actual 
situations. 

Based on this, student assessment of learning in science education 
should encompass context-based situations (using scenario derived from 
real life). 

SOLO taxonomy 

Research has found that despite enhancing scientific literacy through the 
development of multiple competences, a gap exists between students’ 
competences in passing formal assessments based on their knowledge and 
their understanding of its applications in the real world e.g. demonstrating 
scientific literacy in real world situations (Soobard & Rannikmae, 2011; 
OECD, 2014). One possible solution for alleviating this gap has been to pay 
more attention to the student perspective when designing assessment tasks 
for students (Biggs et al. 1991). Here the meaning of student perspective 
was taken to be the knowledge and skills, useful and meaningful for 
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students in their own lives, and applied in a meaningful context for students 
(Feinstein 2010; Fensham and Rennie 2013). This further supported the 
development of scientific literacy, because a major expectation for 
scientific literacy was to ensure that students were capable of applying 
science knowledge and skills gained in real life situations (Holbrook & 
Rannikmae, 2009; Soobard & Rannikmae, 2014). 

According to Biggs (1996), it is important to state the learning 
objectives in terms of what students are expected to be able to do after 
learning and to design assessment tasks to determine how well students are 
progressing in those activities. Biggs proposed the SOLO taxonomy for 
assessing expected learning outcomes at five levels. The first level (pre- 
structural) was not applicable to this study. The remaining four levels were 
(Biggs, 1996; Moseley et al., 2005): 

• Uni-structural - At this level, students can make simple connections, 
but no deeper meaning is presented. Items at this level usually require 
students to name, acquire terminology, and learn simple procedures. 
Learning goals include acquiring terminology, to accomplish the first 
step in mastering a task. For developing items in this level, one piece 
of information coming directly from the question stem (scenario) can 
be used. 

• Multi-structural - At this level, students can make multiple 
connections between bits of information, but relationships are missing. 
Items at this level require students to combine, describe, list, how 
many ways can you ..., what are the main points... Learning goals 
require coverage of “knowing about” and performing algorithms. For 
developing items in this level, two or more discrete and separate pieces 
of information can be used, all contained in the scenario. 

• Relational - At this level, students can provide relationships between 
information provided. Items at this level require students to analyse, 
criticize, argue, justify, explain, apply and relate X with Y. Learning 
goals emphasize understanding, application, problem-solving, 
conceptualizing, reasoning and inquiring. For developing items in this 
level, multiple pieces of information can be used, each directly 
referring to an integrated understanding. 

• Extended abstract - At this level, students can make connections not 
only within the single subject area, but, beyond this, showing the 
ability to generalise and transfer. Items at this level require students to 
hypothesise, reflect, generate and generalise. Learning goals 
emphasise both in-depth understanding, requiring students to theorize 
about a topic, and generalizing to new applications. For developing 
items in this level, an abstract (or general) principle or hypothesis can 
be used, which is derived from the scenario. 
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According to Biggs (1991), uni- and multi-structural levels are 
concerned with the progressive growth of knowledge and skills in a 
quantitative sense, with the last two (relational and extended abstract) with 
qualitative changes in the structure and nature of what is learned. Thus, the 
sequence from pre-through uni- to multi-structural student learning has a 
focus on the accumulated knowledge; in the move from multi-structural to 
relational with the organisation and structuring of knowledge and in the 
shift from relational to extended abstract with how knowledge may be 
theorised and generalised. Therefore, using taxonomy for describing the 
change in operational skills allows considerations to go further than just 
stating outcomes and allows qualitative changes in learning to be shown. 

Methodology 


The Sample 

This was a comparative study - comparing outcomes provided by one group 
of students at the beginning of grade 10 and another at the end of grade 11, 
tested in the same schools and taught by the same teachers. The students 
were not the same. 

As the unit of analysis was the school, a school-related representative 
sample was compiled from students in 44 Estonian schools. Schools were 
grouped based on location (the capital; towns with at least two gymnasiums; 
rural areas). Location was taken into account to ensure that schools in all 
areas had an equal possibility to be involved. After grouping schools based 
on the locations, schools were then ordered based on average national 
examination results. Every 4 th school was chosen and all students from these 
schools were involved in the study. The final data set contained returns from 
1128 10 th grade and 946 11 th grade students. 

The Instrument 

The test instrument provided a contextualised situation, related to 
extraordinary phenomena in nature (e.g. Dead Sea). It required students to 
possess curriculum-related, interdisciplinary science knowledge and skills 
and to transfer these to a new context compared to well-known textbook 
and workbook examples, giving indicators of scientific literacy identified 
through measures of scientific explanation, problem solving and the manner 
in which decisions made were justified. 

The instrument consisted of 8 items, where one item was related to 
each SOLO structural level and three items geared to relational and 
extended abstract respectively. This was done noting that in the PISA 
international test (OECD, 2007), 15 year old students (grade 9 in Estonia) 
performed well above average on structured items, but much less so on 
items related to problems-solving and decision making i.e. SOLO relational 
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and extended abstract levels. Students’ responses were coded using a three- 
point scale (0 - no answer or incorrect answer to 2 - correct answer). An 
overview of the Dead Sea scenario items, levels of SOLO taxonomy, task 
content and skills are presented in table 1. 


Table 1 Overview of student test instruments 


Level 

Biggs (1996) 


Scenario and items 

in the current 




study 




Criteria 

Key words 

Dead 

Sea 

Knowledge 

area 

Component 
of scientific 




items 


literacy 

Uni- 

One obvious 

Name, acquire, 

1 

Plate 

Giving 

structural 

piece of 

information 

terminology 


tectonics 

scientific 
explanation 
(name one 

correct 

explanation) 

Multi- 

structural 

Use two or 
more discrete 

Combine, 
describe, list. 

2 

Atmospheric 

phenomenon 

Giving 

scientific 


and separate 

list the main 



explanation 


pieces of 

information 

points 



(list two 

correct 

explanations) 

Relational 

Use two or 

Analyse, 

3 

Solubility of 

Problem 


more pieces of 

criticise, argue, 

4 

substances 

solving 


information, 
each directly 
related to an 

justify, 
understand, 
apply, relate, 

5 


(interpreting 
and analysing 
graph, 


integrated 

understanding 

explain, solve 
problems, 
inquiry, 
conceptualise 



writing 

scientific 

explanation) 

Extended 

Use abstract 

Hypothesise, 

6 

Heredity 

Decision 

abstract 

general 

reflect, 

7 

Science 

making 


principle or 
hypothesis 

generate, 

generalise, 

depth 

8 

Economic 

(choosing 

correct 

claims. 



understanding, 
theorising 
about the topic, 
generalising 
new 

applications 



writing 

claims, 

making 

justified 

decision) 


To monitor the curriculum related progression, items in the test were 
sequenced so that initial items were less demanding (items in the test 
followed the same sequence as in table 1). This was intended to give 
students the feeling that they could respond to at least some items. The first 
two items were multiple-choice. Item 1 required students to transfer their 
knowledge about plate tectonic to a new context and choose correctly the 
scientific explanation from the four available options to explain what 
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happens in areas around fault lines. Item 2 required students to choose two 
correct scientific explanations related to the formation of fog. 

Items 3-5 introduced a problem-solving situation. In item 3, students 
were asked in interpret data scientifically from a graph of solubility curves 
choosing the first three salts likely to be precipitated under the given 
conditions. In item 4, students were asked to give scientific explanations for 
their choices. In item 5, changed conditions were introduced compared to 
the initial text in item 3 requiring students to give scientific explanations 
for the solubilities of salts at night when the water temperature dropped 
from 35°C to 10°C. 

Items 6-8 focused on extended abstract thinking associated with 
decision-making. In item 6, students were introduced to a situation 
requiring them to choose correctly evidence to support the generalisations 
related to heredity and psoriasis. In item 7, students were required to put 
forward as many claims as they could (by listing) related to increasing the 
trustworthiness of the given study related to psoriasis. In item 8, students 
were required to make a decision, based on the given three options 
(hypothetical economic activities in the area of the Dead Sea) and giving 
argued reasons for their decision. 

The instrument was validated using: 

a) The expert opinions of four school science teachers and two university 
science staff members. Based on their recommendations, the 
instrument was modified to make it more suitable for both the upper 
secondary level and expectations at the university level. 
Considerations related to the University level were included, because 
many upper secondary students were likely preparing to go to 
university. 

b) A pilot study was also conducted on the revised instrument among 
upper secondary school students. 

The reliability, calculated using Cronbach alpha for the overall 
instrument, was 0.62 and considered acceptable for this under ten item test 
instrument. The data gathering period was November 2011 (for grade 10) 
and April/May 2012 (for grade 11). The test was designed to be answered 
in 45 minutes. 

Data analysis 

To explore whether the test instrument followed item distinction, principal 
component factor analysis with Varimax rotation was undertaken. This 
analysis described 68% of the variance and identified a 4 factor solution: 
uni-structural (factor loading 0.975, item 1), multi-structural (factor loading 
0.983, item 2), relational (factor loadings 0.863-0.553, items 3, 4, 5) and 
extended abstract (factor loadings 0.746-0.628, items 6, 7, 8). To ensure 
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that Principal component factor analysis was applicable, a Kaiser-Meyer- 
Olkin Measure of Sampling Adequacy (0.707) and Bartlett’s Test of 
Sphericity were conducted (sig. 0.000). The same factors were found in 
both grades. 

IBM SPSS Statistics 21 was used to describe students’ responses 
frequency distribution and to determine how responses to single item mean 
scores varied between grades by using an Independent Sample T-Test. 
Cohen's d was used to calculate the effect size to eliminate sample size 
influence. 


Results 

Results showed that in both grades (table 2), students achieved at the uni- 
and multi-structural level in a similar manner. For the first two items, there 
was no significant difference between grades in giving scientific 
explanations. The effect size was very small and not taken to be meaningful. 

At the relational level, students in both grades exhibited difficulties. 
To item 3, grade 10 students gave more correct answers (M=0.62) 
compared to grade 11 (M=0.55) and this difference was statistically 
significant (p=0.008), but due to the large sample size, the effect size was 
also calculated; this statistic was found to be small and not taken to be 
meaningful. In items 4 and 5, grade 11 students were generally more able 
to write the scientific explanation on their own than was the case for grade 
10 students. In item 5, there was a statistically significant difference 
between grade 10 and 11 students’ means scores (p=0.000), but the effect 
size remained small and was not taken to be meaningful. 

At the extended abstract level, students in both grades did well in 
responding to item 6 and the response distribution was similar for both 
grades (no statistically significant difference was found, p=0.143). For item 
7, students in grade 11 gave more correct responses compared to grade 10. 
The only item that had a meaningful effect size (as well as the statistically 
significant differences based on t-test) was item 8. However, the mean score 
by students in grade 11 was still relatively small. 

Discussion 

This study investigated the progress in rational scientific literacy skills 
through testing cognition associated with undertaking scientific processes 
and investigated whether curriculum related progress occurred during the 
two years of upper secondary schooling. 

In analysing student responses, based on the grade level at the uni- and 
multi-structural level, percentage distributions between three response 
categories were similar in both grades. 
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Table 2 Frequency distribution and differences in student responses for grades 10 and 11 


SOLO 

level 

Item 

Grade 10 
(N = 1128) 

Grade 11 
(N = 946) 

Independent sample t- 
test (comparison of 
mean scores) 

Effect size 
Cohen’s d 

% 

Mean 

score 

SD 

% 

Mean 

score 

SD 

t 

P 

0 

1 

2 

0 

1 

2 

Uni-str. 

1 

22.2 

28.0 

49.8 

1.28 

0.80 

25.5 

27.3 

47.3 

1.22 

0.82 

1.642 

0.101 

0.074 

Multi-str. 

2 

13.6 

69.2 

17.2 

1.04 

0.55 

12.8 

71.7 

15.5 

1.03 

0.53 

0.370 

0,712 

0.018 

Relational 

3 

40.2 

58.1 

1.8 

0.62 

0.52 

45.7 

53.2 

1.2 

0.55 

0.52 

2.663 

0.008 

0.135 

4 

65.0 

27.7 

7.4 

0.42 

0.63 

63.2 

26.6 

10.1 

0.47 

0.67 

-1,587 

0.113 

0.077 

5 

86.7 

11.8 

1.5 

0.15 

0.39 

81.3 

15.6 

3.1 

0.22 

0.48 

-3.558 

0.000 

0.160 

Extended 

abstract 

6 

5.9 

34.9 

59.2 

1.53 

0.60 

4.5 

33.8 

61.7 

1.57 

0.58 

-1.465 

0.143 

0.067 

7 

73.2 

20.6 

6.2 

0.33 

0.59 

61.0 

25.9 

13.1 

0.52 

0.71 

-6.572 

0.000 

0.291 

8 

69.4 

25.9 

4.7 

0.35 

0.57 

48.6 

47.0 

4.3 

0.56 

0.58 

-8.0.81 

0.000 

0.365 




Items created at the first two SOLO taxonomy levels were more similar to 
traditional testing in science education through examinations and example 
questions in science textbooks. In those items, students were required to 
respond to a clear question, e.g. demonstrating one skill - giving scientific 
explanation, but not to demonstrate multiple skills together. As the results 
were similar in both grades, it could be assumed that there was no 
significant effect size in giving scientific explanations related to those 
SOLO taxonomy levels illustrated by these test items. 

At the relational level, results showed that students at both grades 10 
and 11 found it difficult to solve problems. One reason for this could be that 
in the three items (3-5), students were required to undertake multiple 
activities by themselves. First, they needed to interpret data and evidence 
from the text and graph scientifically and then explain their own choice 
scientifically. Of importance, students needed to make a choice, not given 
in the item and this placed more interpretative responsibility on the students, 
compared to items 1 and 2. Results indicated that if students were required 
to utilize multiple skills together, and hence take more responsibility, 
students faced difficulties. Even more, in the second part of this problem 
solving set (item 5), the salt production conditions were changed (night 
time, temperature drop for the water) and again scientific explanation was 
needed. In this item, students were required to return to the original graph 
and text and again interpret the data and evidence once more. Results 
showed that this type of problem solving was difficult for students in both 
grades. Based on this single scenario, little gain was indicated. A reason for 
this could be that the operational skills, needed for solving such items, were 
not commonly promoted in science subjects in either grade despite the extra 
years of learning and therefore students lacked the necessary skills. In fact, 
as there was no important effect size between grade achievements, it could 
be assumed that no effective teaching and learning occurred in terms of 
scientific literacy, as measured by this study. Possible reasons for this could 
be related with received science teaching. Teachers in Estonia said that their 
self-efficacy was rather low in inquiry-based learning (closely related with 
problem-solving situations) (Holbrook, Rannikmae & Valdmann, 2014). 
Even more, students in both grades involved with this study indicated that 
problem solving was not a perceived focus in their science classes 
(especially in chemistry and physics) (Soobard & Rannikmae, 2014). 

Based on the outcomes from this study, it was apparent that students 
were not being prepared to think rationally (school science vs. real life 
needs) by themselves, and as Rennie (2010) pointed out, science concepts 
in school are simplified, idealised and separated from other affecting factors 
occurring in real life. 

Within the extended abstract items, an interesting outcome was that 
item 6 was shown to be easier for students than items 7 and 8. However, 
extended abstract has its range (Biggs 1996). In item 6, students from both 



grades gave mainly maximal responses. In this item, students were given 
text related to research findings; but rather than being asked to write down 
the correct claims by themselves, students were given possible claims 
related to research outcomes, based on the evidence given in the table, from 
which they were asked to decide the correct claims. This relatively simple 
item, as judged by students, was used in this study, because based on the 
factor analysis, it correlated heavily with the other two extended abstract 
level items focusing on the decision making process. Thus choosing correct 
claims to support the findings was considered as a first step towards 
effective decision-making. However, in item 7, students were asked to write 
single claims by themselves to raise the trustworthiness of the given study 
and item 7 showed, in general, students found writing claims unguided was 
more difficult. The outcomes, based on those two items, indicated that 
students could recognise claims, but did not possess skills to express claims 
on their own, as shown by the low effect sizes when comparing grade 10 
student outcomes with those from grade 11 (table 3). This outcome was 
similar to the set of items on problem solving, where students’ responses 
were not only at lower levels on the three-point scale, but the effect size 
was low if more responsibility on finding the solution was given to students. 

In item 8, students were expected to go beyond stating science 
knowledge and skills and to j ustify a decision made. Students from both 
grades responded poorly to this item and there was almost no difference in 
percentage of correct responses between the grades. The main change 
between grades emerged from the percentage of students at the no or 
incorrect (0) and partially correct (1) response levels (table 3). This type of 
item had not been the focus in the Estonian national examinations in science 
according to students’ perceptions; it was also not a focus in the teaching of 
science subjects (Soobard & Rannikmae, 2014). The findings support the 
idea that assessment in science subjects was strongly related to that valued 
by educators, policy makers and society (Corrigan et al., 2013). There 
seemed to be a mis-match between curriculum expectations on the one 
hand, and on the other, the past national examination thrust and with it, the 
current expectations by science teachers in their teaching. Further research 
was required to elucidate this more specifically. Nevertheless, justified 
decision-making was heavily appreciated as part of scientific literacy in the 
literature (e.g. Choi et al., 2011). A further reason that extended abstract 
thinking appeared not to be taught in science subjects, could be teachers’ 
lack of experience and low self-confidence to include issues arising from 
real life, these being controversial and going beyond a single discipline 
requiring an interdisciplinary understanding about phenomena in nature. 
However, Rennie (2011) stated that this conflict was an outcome because 
education in school was arranged around single disciplines and therefore 
teachers, as well as students, did not promote interdisciplinary components 
in the everyday school context. Nevertheless, according to the Estonian 
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curriculum (Estonian Government, 2011), decision-making was an 
expected learning outcome at the end of grade 12 in all science subjects. 

Results from this study indicated that gymnasium students’ 
demonstrated abilities in operational skills at the uni- and multi-structural 
level, seemingly well developed at pre-gymnasium levels, but poor 
acquisition of relational and extended abstract level skills, even after two 
years of gymnasium study. In general, results showed that responses by 
grade 10 and 11 students, spread similarly between levels for all items 
(particularly for the correct answers), did not exhibit the expected shift for 
grade 11 students, compared to those in grade 10. Regrettably, percentages 
of correct responses were very low for end of grade 11 students, after nearly 
two years of schooling at the upper secondary level; for example, in item 5, 
the maximal student response was only 3.1% (table 3). This suggested there 
was a need to re-think and change the focus of science teaching and the 
emphasis of learning approaches in schools and the manner how science 
subjects were assessed. The impression gained was that examination and 
test items had tended to focus on school science content and its applications 
within science, but not on real life (at least as a rationale for science 
learning). As a result, students were unlikely to receive feedback about their 
success in applying science in everyday situations and their only action was 
to apply school science in a school science context. Even more, teachers 
were not in a position to support students, as they were not receiving 
feedback on students’ actual progress (Corrigan et al., 2013). Some may 
feel the limited number of test items (8) in the current study is a weakness, 
but findings indicate that it is sufficient to develop substantive findings for 
this study. Furthermore, this study used representative sample to implement 
the instrument and draw conclusions. 

Conclusions 

The main outcomes showed that despite two years of schooling, grade 10 
and 11 students exhibited little effect size gains in components of scientific 
literacy. Even though, students exhibited a small yet noticeable 
improvement, for operations involving higher levels of SOLO taxonomy 
such as making reasoned decisions and solve scientific problems, outcomes 
were very poor. 

Outcomes showed that grade 10 and 11 students demonstrated their 
operational scientific literacy skills situations in a similar manner when 
responding to context-based test items, suggesting little teaching was taking 
place in these aspects of learning. 
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Implications and Recommendations 


This outcome must be of concern pointing to a serious need to re-assess the 
teaching and learning expected at the end of grade 11. 

Based on the outcomes from this study, the following 
recommendations are put forward for science teaching: 

1. To overcome the difficulties related to school science simplifications, 
idealisations and the fact that science is separated between disciplines, 
more real life related issues and interdisciplinary approaches need to 
be used in science subjects and examinations. 

2. Based on the outcomes of this study, there is a need to focus more on 
learning progression in operational scientific skills development. This 
can support the situation where more students are capable to solve 
scientific problems in their lives and make reasoned decisions in 
everyday life situations. 

3. Science teachers need to take interested in finding out students’ 
perceptions and be ready to modify their science teaching based on the 
findings, e.g. supporting students when needed (giving more guidance, 
analysing students’ difficulties). However, these activities are also 
related with science teachers’ own perceptions against themselves and 
their science teaching. 

4. Teachers need to re-consider the way science is presented and 
assessed. The focus of assessment in science subjects needs to move 
towards including real life contexts and actual operational skills 
leading towards higher levels of scientific literacy. There is also a need 
to investigate student’s progress from moving one grade to another (a 
longitudinal study) to give insight about students’ progress. 

Limitations 

This study compared student’s outcomes from grade 10 and 11. Although 
studying in the same schools and taught by the same teachers, the students 
were different. Therefore, it was not possible to make conclusions about the 
development of specific students from one grade to another. 

The number of test items used to develop the substantive findings in 
this study, for each of the SOLO categories, were limited and only one 
specific context was used to develop the questions asked. 
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Appendix 


Appendix Relating scientific literacy components in this study with the multiplicity of definitions in the literature 


Components 
associated with 
scientific literacy 

NRC, 1996 

Norris and 

Phillips, 2003 

OECD, 2007 

Holbrook and 
Rannikmae, 

2009 

Choi et al., 

2011 

Components of 
scientific literacy 
used in this study 
for item 

development based 
on the relevant 
literature 

Science knowledge 
and application 

Knowledge and 
understanding of scientific 
concepts, facts, principles, 
laws, theories. 

Understanding 
basic scientific 
ideas. 

Scientific 
knowledge 
and use of this 
knowledge. 

Utilising 

appropriate 

evidence-based 

scientific 

knowledge. 

Inter¬ 
disciplinary 
under-standing 
of science 
knowledge and 
use if this 
knowledge. 

Application of 
(interdisciplinary) 
science knowledge 
in real life related 

context. 

Operational skills 

Ability to make informed 
personal decisions. Ability 
to pose/evaluate arguments 
based on evidence and to 
draw conclusions. Ability 
to describe, explain and 
predict natural phenomena. 
Ability to use inquiry 
process (e.g. to evaluate 
the quality of scientific 
information based on its 
source and methods). 

Applying scientific 
knowledge for 
problem solving; 
participating in 
science-based 
social issues. 

1. Explaining 
phenomena 
scientifically. 

2. Using 
scientific 
evidence. 

3. Identifying 
scientific 
issues. 

Solving 

meaningful 

scientific 

problems, 

making 

reasoned socio- 

scientific 

decisions. 

Competencies 
to solve 
problems, 
reasoning, 
finding and 
using resources, 
applying core 
ideas, arguing 
for and against 
based on 
evidences. 

Use of scientific 
literacy cognitive 
operational skills: 

1. Explaining 
phenomena 
scientifically. 

2. Problem solving 
based on real life 

context. 

3. Decision making 
and reasoning based 
on real life context. 




Appendix Relating scientific literacy components in this study with the multiplicity of definitions in the literature (Cont.’d) 


Nature of Science 
(NOS) 

Understanding about 
science and its interactions 
to society. 

Understanding 
science and its 
applications; 
knowledge what 
counts as science; 
understanding 

NOS. 

Knowledge 
about science. 


Understanding 

nature of 

science (e.g. 

testable, 

creative, 

theory-laden, 

evidence). 

These components 
(NOS and 
perceptions) while 
recognized as 
components of 
scientific literacy, 
are not emphasised 
in the current study 
which focuses on the 
student’s 
operational skills 
and application of 
science knowledge. 

Perception 
components 
(attitudes; values; 
interest, views 

towards one self, 
career and science, 
etc.) 

Appropriate attitudes and 
values towards science. 

Lifelong learner; 
participating in 
science related 
issues. 

Appreciation of 
science, curiosity. 

Attitudes 

towards 

science (e.g. 

interest, 

support, 

responsibility). 


Beliefs, 

understanding 

one’s own 

cognitive 

abilities, 

appreciating 

life-long 

learning. 
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